1. Technical Field
The present invention relates to a memory aid method of selecting data watched by a user, recording the selected data along with situation/behavior data upon watching, extracting recorded audio/video data in response to a search instruction from a user or a situation condition, and providing the extracted audio/video data.
2. Description of the Related Art
As technology advances, modern life requires more skills and knowledge, and thus more recognition abilities are required. Furthermore, more jobs that require more recognition abilities are generated compared to the past, and a person's power of memory is overwhelmed by overflowing information in modern life. In particular, persons are being buried with enormous information poured out from the Internet, smart phones, tablet computers, TV, and radios.
One of solutions to such an information overload is a memory aid system. As if a vehicle has expanded the mobility of a human being, a memory aid system may support a human being so that a person's power of memory can be increased and a human being can further show its ability in the information overload society. Such a memory aid system needs to continue to operate along with human memory. That is, the memory aid system needs to understand the operation method of the human memory, know the strength and limits of the human memory, and instantly provide accurate information at a proper Lime.
The human being's memory may be divided into prospective memory (PM) and retrospective memory (RM). PM is defined as the ability to remember to remember (Winograd, 1988). For example, PM refers to memory that a call is to be made to a friend on Friday next week. RM refers to memory that a thing happened in the past.
Problems in PH widely appear in the aged in everyday life. For example, an aged person frequently forgets the time when pills are taken or a lunch time. The problems in PM may be easily solved. Lots of apps for smart phones that provide notification of things to do on a scheduled time are being developed [Patent Document 1].
In contrast, a memory aid technology in RM includes many problems to be solved, and related research is in progress.
For example, a system architecture used in the HERMES project has been suggested as research that suggests architecture for analyzing data collected by sensors and transmitted to a server [Non-patent Document 1]. Such a system includes cameras installed indoors and outdoors. Perceptual components responsible for a visual analysis and auditory processing extract metadata by processing audio/video input received through the cameras, and the extracted metadata is stored in the database (DB) of the system. Application data and the metadata are stored in two separate repositories. The first repository is a relation DB and used to store the application data. The second repository is a knowledge base repository and is used for tasks for verifying the validity of the metadata, inferring new knowledge, and applying a rule based on a rule engine.
For example, SenseCam of Microsoft, that is, an apparatus in which a wearable device continues to record images, is an apparatus for storing a series of images and sensor data when a user makes a living while wearing the apparatus. It was revealed that an amnesiac's memory can be helped by periodically showing images recorded by the apparatus to the amnesiac [Non-patent Document 2]. For another example, it was revealed that memories in the life of a patient having damaged memory can be improved by such a method [Non-patent Document 3]. Such research revealed that memory improvement was helped by showing continuously recorded images, but have difficulties in using the research in real time because the amount of provided video data is too much. Accordingly, there is a need to allow a user to select important images from collected video data.
Furthermore, it was revealed that anecdote memory can be improved by suggesting clues, such as persons, behaviors, subjects, and places, to a patient having damaged anecdote memory [Non-patent Document 4]. Such research results suggest a possibility that a user's anecdote memory may be improved using information, such as persons, behaviors, subjects, and places that have been extracted from continuously input audio/video data and stored.
Furthermore, a patent document suggests a 3-layered architecture for a memory aid [Patent Document 2]. A small wireless sensing device is dynamically programmed, and automatically detects important information related to the current context and environment of a wearer. The wireless sensing device determines whether specific information is important for a user using an internal rule engine and uploads the important information to a mobile application. The mobile application analyzes the important information using other information obtained by sensors and uploads a complete information set to an Internet service application. The Internet service application processes the uploaded information using another external data source and an internal processing engine and transfers the processed uploaded information to a user through a notification engine.